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Abstract 

This paper describes the results of an evaluation of a self-paced BCI application conducted 
with 21 naive subjects. We studied both performances and preferences of subjects placed 
voluntarily in a challenging situation: first-time session, no human learning, no machine 
learning of the mental state to be detected, “out of the lab”, use of a single EEG channel. 
The application consisted of an entertaining interaction with a virtual world inspired by the 
famous “Star Wars” movie. Subjects were asked to control the take-off and height of a virtual 
spaceship by using their motor-related brain activity. Results showed that, without training, 
roughly half of the subjects were able to control the application by using real foot movements 
and a quarter were able to control it by using imagined foot movements. Taken together, 
the results of subjective questionnaires stressed the importance of mental strategies and the 
need to provide subjects with a continuous and complete visual feedback, even when the 
non-control state is detected. In addition, the whole application appeared as enjoyable and 
motivating for the subjects. 


1 Introduction 

A lot of Brain-Computer Interfaces (BCI) studies are conducted inside laboratories, in highly 
controlled conditions, and with a relatively small number of subjects trained over a number of 
sessions which may be large. Furthermore, most current studies are focused on the BCI perfor¬ 
mances and not on the subjects’ preferences. A notable exception is the work of Guger et al which 
has evaluated a BCI with 99 naive subjects during an exposition [I]. This work focused on the 
performances of subjects who had to control a synchronous, 2-class BCI based on 2 bipolar EEG 
channels and a trained classifier. Subjects were asked to imagine movements of their right-hand 
or their feet and were provided with a simple 2D visual feedback. Their results showed that 93% 
of the subjects were able to reach an accuracy equal or greater than 60%. 

In this paper, we study both the performances and the preferences of 21 naive subjects during 
an exhibition. These subjects used a self-paced (asynchronous) BCI, based on a single EEG 
channel, which does not use machine learning of the mental state to be detected. The subjects 
could interact with an entertaining virtual reality application inspired from the “Star wars” movie. 


2 Method 

2.1 The BCI system 

We have designed a simple self-paced BCI system based on real or imagined foot movements. This 
BCI is based on a single EEG channel, located at position Cz and aims at detecting a Beta Event 
Related Synchronisation (ERS), appearing posterior to the real or imagined foot movement [2]. 
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To this end, the EEG signal is first band-pass filtered in the 3-45 Hz band. Then, a Band Power 
(BP) feature [2] is extracted in the Beta band (16-24 Hz) for the last second of data. This feature 
is extracted every 100 ms and the last four consecutive features are averaged (with a moving 
average) in order to produce a smooth Control Signal (CS). 

To detect the Beta ERS, and hence, the foot movement, we used a simple threshold Th. If 
the computed CS was higher than this threshold Th, a foot movement was detected (intentional 
control state) and a command was sent to the application. If the CS was lower than the threshold 
Th, the non-control state was detected and no command was sent to the application. The value 
oi Th was simply determined according to the mean fj, and standard deviation tr of a CS epoch 
obtained while the subject was relaxed, according to Equation 1. 

Th = ^ + 3a (1) 

2.2 The Virtual Reality application: “Use the force!” 

We have developed an entertaining Virtual Reality (VR) application, as Leeb et al have shown 
that VR can increase the motivation and interest of subjects during BCI experiments [3]. Our 
virtual environment corresponds to a “Star Wars” space vessel, in which the subject could see 
a virtual spaceship (a Tie-Fighter) and a static character (Darth Vader) (see Figure 1). The 
purpose of the application was to lift the Tie-Fighter up by using the BCI. This task established 
an analogy between the use of the BCI and the use of “the force” in the Star Wars movie. As 
such, the application was named “Use the force!”. More precisely, the Tie-Fighter was lifted-up 
when the VR application received the corresponding command from the BCI. The Tie-Fighter 
was lifted-up at a speed proportional to the value of the CS. When no command was received, the 
Tie-Fighter went down. 

The VR application was developed with the OpenMASK VR platform [4] and the BCI was de¬ 
veloped with the OpenViBE BCI platform^. The VR application and the BCI system were easily 
connected using the VRPN protocol, thanks to the dedicated modules of OpenViBE. 

2.3 The experiment 

Subjects participated in an experiment which lasted about 45 minutes. This experiment was 
divided into seven succesive steps: 

1- Electrode montage: Only three electrodes were used: a ground electrode (located on the 
forehead), a reference electrode (located on the nose) and the Cz electrode. Electrode Cz was 
fixed using an adhesive paste instead of a cap, for a faster setup. 

2- Signal visualization: Subjects were shown their EEG signal recorded at Cz (band-pass fil¬ 
tered in 3-45 Hz) while they were clenching their teeth or blinking. This aimed at showing them 
the need to be as relaxed as possible during the experiment and the need to avoid blinking. During 
the next steps, subjects were regularly reminded to stay as relaxed as possible. 

3- Baseline: During this step, subjects were asked to stay relaxed. Once they were relaxed, 20 
seconds of EEG signal were recorded and converted into a CS which was used to compute Th 
using Equation I. 

4- Free time: During this step, the subject could interact freely with the VR application by 
using real foot movements. When the BCI detected a Beta ERS, the Tie-Fighter was lifted- 
up. Alternatively, the CS was shown to the subject so that he could see the impacts of real 
foot movements on the Beta power. This step aimed at making the subject familiar with the 

^ WWW. irisa.fr/bunraku/OpenViBE 
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application and with the task. If the user seemed unable to lift the spaceship, the baseline step 
was performed again, in order to obtain a new Threshold Th. Then, the next steps followed. 

5-Real movement game: During this game, subjects had to lift the Tie-Fighter up, by perform¬ 
ing real foot movements during specific periods instructed by the application. These instructions 
are used to evaluate the system but are not used by the BCI for classifying the input data. Actu¬ 
ally, the user can lift the Tie-Fighter up at any time and all the game long, independantly from 
the instructions. In other words, we used a ’’paced test environment” to evaluate this self-paced 
BCI [5]. The game was composed of 10 trials. Each trial lasted 10 s, and was divided in three 
phases (see Figure 1): 1) A resting phase lasting 4 seconds during which no specific task was given 
to the subject. 2) A “move” phase, lasting 3 seconds, during which the subject was instructed to 
perform real foot movements. The instruction was given using a green text “move” appearing on 
the screen. 3) A “stop” phase lasting 3 seconds, during which the subject was instructed to stop 
performing the movement in order to lift the Tie-Fighter up. The instruction was given using a 
red text “stop moving” appearing on the screen. If the subject managed to lift the Tie-Fighter 
up during this third phase, his score was increased and displayed using a yellow gauge located on 
the left corner down the screen. 
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Figure 1: Temporal structure of a trial of the VR game. 


6- Imagined movement game: This game was identical to the previous one except that sub¬ 
jects were instructed to perform imagined foot movements instead of real foot movements. We 
instructed subjects to perform kinaesthetic motor imagery rather than visual motor imagery. 

7- Questionnaire: After the experiment, subjects were asked to fill in a subjective questionnaire. 

3 Results 

The experiments took place during the Laval Virtual 2008 VR exhibition, on a booth. As such, the 
environment was a noisy environment with persons moving and talking around. 21 naive subjects 
(mean age: 33.48±9.14), 18 males and 3 females, participated voluntarily to the experiment. No 
selection was performed and all volunteers were accepted. All subjects gave their written informed 
consent before the experiment. 

3.1 Subjects’ performances 

We assessed the subjects’ performances by computing the number of True Positives (TP) and False 
Positives (FP) they obtained during the games [5]. We counted a single TP when the CS value 
became higher than the threshold Th once or more times during the “stop moving” phase (see 
section 2.3). We counted a single FP when the CS value became higher than the threshold Th once 
or more times during the “move” phase, during which a Beta Event Related Desynchronisation 


3 



(ERD) should be observed and not a Beta ERS. What happened during the resting phase was not 
taken into account in the performance analysis. From the FP and TP, we computed the Hit-False 
(HF) difference, which corresponds to the number of TP minus the number of FP [o]. Performances 
obtained by subjects are summarized in Figure 2, under the form of absolute frequency diagrams 
for TP and HF difference. They show the number of subjects who obtained a given performance, 
for real and imagined movement games separately. 
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Figure 2: Absolute frequency diagrams for TP and HF difference, for real or imagined movements. 

These diagrams shows that about half of the subjects (12 subjects out of 21) reached an HF 
difference > 3 using real movements, and that about a quarter (5 subjects out of 21) reached an 
HF difference > 3 using imagined movements. According to simulations performed as described 
in [6], a system which reached an HF difference > 3 with 10 trials per class, is better than a 
randomly performing system (one-tailed test) with a probability of type I error < 0.054. This 
suggests that, roughly half of the subjects had at least a small control over the Tie-Fighter using 
real movements and that a quarter had at least a small control using imagined movements. The 
mean HF difference was 3.14 ± 2.24 for real movements and 1.33 ± 2.03 for imagined movements 
while the mean TP was 4.95 ± 2.18 for real movements and 2.67 ± 2.08 for imagined movements. 
These results may appear as modest but one should consider the fact that subjects were naive and 
untrained and that a very simple BCI design was used. Actually, we used a single EEG channel, 
placed at a standard location (i.e., a non-optimized location) and we used a single feature, based 
on a standard frequency band (i.e., a non-optimized frequency band), with a simple threshold. 
In future works, it would be interesting to study subjects’ performances while using different 
frequency bands and electrode locations. 

3.2 Subjective questionnaires 

3.2.1 Quantitative data 

Subjects were asked to grade questions by giving a mark between 1 and 7. Table 1 displays the 
average marks given by the subjects for the two conditions (real movement game and imagined 
movement game) according to various criteria. 

Our results first showed that the experiments did not seem tiring for the subjects. The ex¬ 
periments with imagined movements seemed however more tiring than that with real movements. 
However, this difference is not statistically significant (Wilcoxon test W = —28, p > 0.1). Despite 
the use of pastes to fix the electrodes, subjects found the experiment comfortable (global mean for 
question 2: 5.19±1.23). According to oral discussions with subjects, it seemed that their curiosity 
and will to test a BCI was stronger than their apprehension to have gel in their hair. 

Concerning the control, it seems that subjects felt to have an average control over the space¬ 
ship using real movements whereas they felt to have a lower control using imagined movements. 
As expected, subjects had significantly more trouble controlling the spaceship using imagined 
movement than using real movements {W = 79, p < 0.01). Clobally, subjects were able to assess 
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Question 

Answer for 
real movements 

Answer for 
imagined movements 

1- Did you get tired because of the experiment? 

(1: not tired at all, 7: very tired) 

1.76 ± 1.04 

2.05 ± 1.40 

2- Did you find the experiment comfortable? 

(1: not comfortable at all, 7: very comfortable) 

3- Did you feel that you could control the spaceship 

5.10 ± 1.26 

5.29 ± 1.23 

(that is that you could lift it voluntarily?) 

(1: you didn’t feel you could control it at all, 

7: you controlled it perfectly) 

4- Did you feel frustration or annoyance during the 

3.95 ± 1.80 

2.81 ±1.86 

experiment? (1: no frustration or annoyance, 

7: a lot of frustration and annoyance) 

2.33 ± 1.56 

3.29 ± 1.65 


Table 1: Average marks given by the subjects in the questionnaire, for the two conditions. 


properly their performances, as the marks they gave for question 3, related to their feeling of con¬ 
trol, are significantly correlated with the HF differences obtained (Spearman correlation = 0.63, 
p < 0.00001). Concerning only imagined movement games, the marks given by subjects to ques¬ 
tion 3 are significantly correlated with both the HF difference and the TP rate they obtained 
{p < 0.05). Interestingly enough, this correlation is slightly higher between the marks and the TP 
rate (rg = 0.56, p < 0.01) than between the marks and the HF difference (r^ = 0.51, p < 0.05). 
This is not the case for real movement games for which there is no correlation between the marks 
and the TP obtained (r^ = 0.31, p > 0.05). This might suggest that for a difficult task such as 
lifting the spaceship using imagined movements, subjects paid more attention to the fact that the 
spaceship went up when it should have (TP) than when it should not have (FP). 

Finally, questionnaire answers showed that subjects found real movement games not really 
frustrating or annoying whereas imagined movement games where more frustrating and annoying. 
The difference between the two conditions is significant (W = —64, p < 0.05). This frustration 
might be due to the inscreased difficulty to lift the spaceship with imagined movement. However 
and surprisingly, there is no correlation between the frustration felt by subjects during the imagined 
movement games and their performance, i.e., the HF difference they obtained, (r^ = —0.11, 
p > 0.05) nor between the frustration felt and the subject impression of control (rg = 0.26, 
p > 0.05). One explanation could be related to the absence or lack of visual feedback. Indeed, 
during imagined movement games, subjects had generally fewer feedback as the spaceship was 
lifted up less often or it was lifted up less high and stayed in the air a shorter time. This may 
suggest that less feedback leads to more frustration, whatever the performance. This seemed to 
be confirmed by oral discussions with subjects. 

3.2.2 Qualitative data 

Thanks to the use of open questions, the questionnaire enabled us to investigate which kinds of 
imagined movements the subjects performed, as well as to obtain their remarks and comments 
concerning the application itself. 

Regarding the kinds of movement imagined by the subjects, it is interesting to note that a 
large variety of strategies where employed. For instance, subjects reported that they imagined 
themselves swimming, running, taping their feet, braking and accelerating, walking or using stairs. 
7 subjects reported they imagined the same foot movement that the one they did in the real 
movement game, whereas 10 reported they imagined a different movement. On average, subjects 
for whom the real and imagined movements were the same obtained better results (mean HF=3 ± 
2.67) than the others (mean HF=0.8± 1.75). However, this difference is not statistically significant 
(Mann-Whitney test C/7,10 = 20, p = 0.16), but it would be interesting to study this point further 
in the future, by using a dedicated experiment. Interestingly, 13 subjects reported they used a 
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single strategy during the experiments whereas 7 reported they used several strategies. However, 
there is no difference between these two groups in terms of performances (C/ 13,7 = 41, p = 0.75). 

Concerning the free remarks of subjects, it is interesting to note that 3 subjects complained 
about the difficulty to concentrate considering the environment they were in, i.e., an exhibition. 
They would have prefered to be in a more isolated place. Most subjects reported that they found 
the application and the interface well designed, enjoyable and motivating. These remarks are in 
line with previous studies that showed that VR could increase the motivation of subjects for BCI 

[3]. Finally, another valuable comment made by 2 subjects concerned the frustration they felt 
due to the absence or lack of feedback when they did not succeed to lift the spaceship. They 
suggested that an additional or more complete feedback could be used in order to give them more 
information and, possibly, improve their learning. 


4 Discussion and conclusion 

This paper^ described the results of an evaluation of a self-paced BCI application conducted with 
21 naive subjects. We studied both performances and preferences of subjects placed voluntarily in 
a challenging situation: first-time session, no human learning, no machine learning of the mental 
state to be detected, “out of the lab”, use of a single EEC channel. Subjects interacted with an 
entertaining Virtual Reality application and were asked to control the take-off and height of a 
virtual spaceship by using real or imagined movements. Results showed that, without training, 
roughly half of the subjects were able to control the application by using real foot movements, 
and a quarter were able to control it by using imagined foot movements. 

Taken together, the results of the subjective questionnaire stressed the importance of the mental 
strategies and the visual feedback. More precisely, results suggested that a lack or an absence of 
feedback during the detection of the non-control state could lead to an increased frustration for 
the subjects. Thus, when designing a self-paced BCI, we recommend to provide subjects with a 
continuous feedback, and to provide feedback (possibly a different one) even when the non-control 
state is detected. For instance, we could imagine a feedback indicating the subject how close he is 
from the intentional control state. This may be likely to reduce the subject frustration, to improve 
his motivation and possibly accelerate his learning. 


References 

[ 1 ] C. Guger, G. Edlinger, W. Harkam, I. Niedermayer, and G. Pfurtscheller. How many people 
are able to operate an EEG-based brain-computer interface (BCI)? IEEE Trans, on Neural 
Sys. and Rehab., II(2):I45-147, 2003. 

[2] G. Pfurtscheller and F. H. Lopes da Silva. Event-related EEG/MEG synchronization and 
desynchronization: basic principles. Clinical Neurophysiology, 110(11):1842-1857, 1999. 

[3] R. Leeb, R. Scherer, D. Friedman, F. Lee, C. Keinrath, H. Bischof, M. Slater, and 
G. Pfurtscheller. Towards brain-computer interfacing, chapter Combining BCI and Virtual 
Reality: Scouting Virtual Worlds. MIT Press, G. Dornhege, JdelR. Millan et al edition, 2007. 

[4] D. Margery, B. Arnaldi, A. Chauffaut, S. Donikian, and T. Duval. Openmask: Multi-threaded 
or modular animation and simulation kernel or kit: a general introduction. In VRIC, 2002. 

[5] S. Mason, J. Kronegg, J. Huggins, M. Fatourechi, and A. Schloegl. Evaluating the performance 
of self-paced BCI technology. Technical report, Neil Squire Society, 2006. 

[6] G. Miiller-Putz, R. Scherer, C. Brunner, R. Leeb, and G. Pfurtscheller. Better than random: 
a closer look on BCI results. International Journal of Bioelectromagnetism, 10(l):52-55, 2008. 

^This work was supported by the French National Research Agency within the Open-ViBE project and grant 

ANR05RNTL01601. Authors also thank Dr. Virginie Attina, Dr. Guillaume Gibert, Dr. Emmanuel Maby and 

Bruno Renier for their help and support related to this work. 


6 



